Ar display with adjustable stereo overlap zone

ABSTRACT

A system and method are disclosed for use in a virtual reality environment including a head mounted display device and a processing unit. In examples, the processing unit adjusts an amount by which left and right displayed images overlap each other at a given distance, such as the focal distance, from the head mounted display device.

BACKGROUND

Augmented reality is a technology that allows virtual imagery to bemixed with a real-world physical environment. A see-through, headmounted display device may be worn by a user to view the mixed imageryof real objects and virtual objects displayed in the user's field ofview. In examples, microdisplays for both the left and right eyesproject images including virtual objects into left and right opticallens assemblies. The optical lens assemblies then project the imagesinto the left and right eyes. By providing the two images with binoculardisparity, the head mounted displays can create a stereopsis effectwhere the virtual images appear three-dimensional at varying depths.

Given the spacing between the left and right optical lens assemblies, ahorizontal stereo overlap of the left and right displayed images varieswith a distance away from the lens assemblies. Lens assemblies may beconfigured to have complete image overlap at a given focal distance (forexample at infinity), but this results in incomplete image overlap atother focal distances. Incomplete image overlap can have some disturbingvisual effects, such as dim bands at the horizontal edges and possible areduced stereopsis effect.

SUMMARY

Embodiments of the present technology relate to a system and method forpresenting three-dimensional virtual objects within an augmented realityenvironment. A system for creating virtual objects in general includes asee-through, head mounted display device coupled to at least oneprocessing unit. The head mounted display device includes a pair ofmicrodisplays and optical lens assemblies, one each for the left andright eyes, for presenting images to the left and right eyes withstereopsis effect.

A focal distance of the left and right lens assemblies may be set to apredefined position during manufacture, such as for example 2 metersfrom the lens assemblies. Thereafter, an algorithm may shift the pixelsat which the microdisplays project the left and right images to providestereo overlap of the left and right images at the focal distance. Themicrodisplays may be provided with spare, generally unused pixels at aborder of the display for use when shifting the displayed left and rightimages.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of a virtual reality environment includingreal and virtual objects.

FIG. 2 is a perspective view of one embodiment of ahead mounted displayunit.

FIG. 3 is a side view of a portion of one embodiment of a head mounteddisplay unit.

FIG. 4 is a block diagram of one embodiment of the components of a headmounted display unit.

FIG. 5 is a block diagram of one embodiment of the components of aprocessing unit associated with a head mounted display unit.

FIG. 6 is a top view of a pair of optical lens assemblies having aboresight focused at infinity.

FIG. 7 is a top view of a pair of optical lens assemblies focusing leftand right images at a focal plane.

FIG. 8 is a top view of a pair of optical lens assemblies displaying avirtual object.

FIG. 9 is a view of a left image used to generate the virtual objectdisplayed in FIG. 8.

FIG. 10 is a view of a right image used to generate the virtual objectdisplayed in FIG. 8.

FIG. 11 is a conventional top view of a pair optical lens assembliesdisplaying left and right images partially overlapped at a number offocal distances.

FIG. 12 is a conventional graph of an amount of horizontal overlapversus distance for two different interocular distances.

FIG. 13 is a flowchart illustrating the operation of embodiments of thepresent technology.

FIG. 14 illustrates left and right fields of active pixels, includingpixels for displaying the left and right image, and a border of pixelsaround the pixels used to display the left and right images.

FIG. 15 illustrates left and right fields of active pixels, includingpixels for displaying the left and right image which have been shiftedinto the border of pixels around.

FIG. 16 is a top view of a pair optical lens assemblies displaying leftand right images overlapped at a focal distance.

FIG. 17 is a flowchart illustrating the operation of a head mounteddisplay device and processing unit according to embodiments of thepresent technology.

FIG. 18 shows further detail of step 614 of FIG. 17 for determining afield of view.

DETAILED DESCRIPTION

Embodiments of the present technology will now be described withreference to the figures, which in general relate to a system and methodfor controlling the overlap of images displayed to the left and righteyes in an augmented reality system. In embodiments, the system andmethod may use a mobile augmented reality assembly to generate athree-dimensional augmented reality environment. The augmented realityassembly includes a mobile processing unit coupled to a head mounteddisplay device (or other suitable apparatus) having left and rightmicrodisplays and optical lens assemblies.

The lens assemblies are to a degree transparent so that a user can lookthrough the lens assemblies at real world objects within the user'sfield of view (FOV). The microdisplays and lens assemblies also providethe ability to project virtual images into the FOV of the user such thatthe virtual images may also appear alongside the real world objects. Thesystem automatically tracks where the user is looking so that the systemcan determine where to insert a virtual image in the FOV of the user.Once the system knows where to project the virtual image, the image isprojected to the left and right eyes with binocular disparity to create3D virtual objects alongside real world objects.

The lens assemblies may be configured with a particular focal distance,which is the distance at which the left optical lens assembly focusesimages, and the distance at which the right optical lens assemblyfocuses images. A focal plane 2 meters in front of the head mounteddisplay device is one example providing a minimum of eye strain. Asnoted in the Background section, given the spacing and relativeorientation of the respective left and right optical lens assemblies,the stereo horizontal overlap of the left and right images will varywith a distance away from the lens assemblies. In accordance withaspects of the present technology, the displayed positions of the leftand right images may be controlled using an algorithm, referred toherein as a display shift algorithm, for maximum horizontal stereooverlap at a desired depth, for example at the configured focaldistance. It is also contemplated that the vertical stereo overlap ofthe left and right images be controlled with the display shiftalgorithm. The optical lens assemblies and the operation of the displayshift algorithm are explained below.

FIG. 1 illustrates an augmented reality environment 10 for providing anaugmented reality experience to users by fusing virtual content 21 withreal content 23 within each user's FOV. FIG. 1 shows two users 18 a and18 b, each wearing a head mounted display device 2, and each viewing thevirtual content 21 adjusted to their perspective. It is understood thatthe particular virtual content shown in FIG. 1 is by way of exampleonly, and may be any of a wide variety of virtual objects.

As shown in FIG. 2, each head mounted display device 2 may include or bein communication with its own processing unit 4, for example via aflexible wire 6. The head mounted display device may alternativelycommunicate wirelessly with the processing unit 4. In furtherembodiments, the processing unit 4 may be integrated into the headmounted display device 2. Head mounted display device 2, which in oneembodiment is in the shape of glasses, is worn on the head of a user sothat the user can see through a display and thereby have an actualdirect view of the space in front of the user. More details of the headmounted display device 2 and processing unit 4 are provided below.

Where not incorporated into the head mounted display device 2, theprocessing unit 4 may be a small, portable device for example worn onthe user's wrist or stored within a user's pocket. The processing unit 4may include hardware components and/or software components to executeapplications such as gaming applications, non-gaming applications, orthe like. In one embodiment, processing unit 4 may include a processorsuch as a standardized processor, a specialized processor, amicroprocessor, or the like that may execute instructions stored on aprocessor readable storage device for performing the processes describedherein. In embodiments, the processing unit 4 may communicate wirelessly(e.g., WiFi, Bluetooth, infra-red, or other wireless communicationmeans) to one or more remote computing systems. These remote computingsystems may including a computer, a gaming system or console, or aremote service provider.

The head mounted display device 2 and processing unit 4 may cooperatewith each other to present virtual objects 21 to a user in an augmentedreality environment 10. The details of the mobile head mounted displaydevice 2 and processing unit 4 which enable the building of virtualobjects will now be explained with reference to FIGS. 2-6.

FIGS. 2 and 3 show perspective and side views of the head mounteddisplay device 2. FIG. 3 shows only the right side of head mounteddisplay device 2, including a portion of the device having temple 102and nose bridge 104. Built into nose bridge 104 is a microphone 110 forrecording sounds and transmitting that audio data to processing unit 4,as described below. At the front of head mounted display device 2 isroom-facing video camera 112 that can capture video and still images.Those images are transmitted to processing unit 4, as described below.

A portion of the frame of head mounted display device 2 will surround adisplay that includes a pair of lens assemblies, each including one ormore lenses. In order to show the components of head mounted displaydevice 2, a portion of the frame surrounding the display is notdepicted. The display includes a pair of optical lens assemblies 113(one of which is seen in FIG. 3) including a light-guide optical element115, opacity filter 114, see-through lens 116 and a focal plane lens118. The left and right focal plane lenses 118 (one of which is seen inFIG. 3) may be behind and aligned with light-guide optical element 115.The focal plane lenses 118 may be negatively or concavely shaped tofocus light received from the light-guide optical element 115 at focaldistance at some predefined distance in front of the optical lensassembly 113 as explained below.

In one embodiment, opacity filter 114 is behind and aligned withsee-through lens 116. Opacity filter 114, which is aligned withlight-guide optical element 115, selectively blocks natural light,either uniformly or on a per-pixel basis, from passing throughlight-guide optical element 115. Details of an example of opacity filter114 are provided in U.S. Patent Publication No. 2012/0068913 to Bar-Zeevet al., entitled “Opacity Filter For See-Through Mounted Display,” filedon Sep. 21, 2010.

In one embodiment, light-guide optical element 115 is behind and alignedwith opacity filter 114. Light-guide optical element 115 transmits lightfrom microdisplay 120 to the eye 140 of the user wearing head mounteddisplay device 2. Light-guide optical element 115 also allows light fromin front of the head mounted display device 2 to be transmitted throughlight-guide optical element 115 to eye 140, as depicted by arrow 142,thereby allowing the user to have an actual direct view of the space infront of head mounted display device 2 in addition to receiving avirtual image from microdisplay 120. Thus, the walls of light-guideoptical element 115 are see-through.

Light-guide optical element 115 includes a first reflecting surface 124(e.g., a mirror or other surface). Light from microdisplay 120 passesthrough lens 122 and becomes incident on reflecting surface 124. Thereflecting surface 124 reflects the incident light from the microdisplay120 such that light is trapped inside a planar substrate comprisinglight-guide optical element 115 by internal reflection. After severalreflections off the surfaces of the substrate, the trapped light wavesreach an array of selectively reflecting surfaces 126. Note that onlyone of the five surfaces is labeled 126 to prevent over-crowding of thedrawing. Reflecting surfaces 126 couple the light waves incident uponthose reflecting surfaces out of the substrate into the eye 140 of theuser.

As different light rays will travel and bounce off the inside of thesubstrate at different angles, the different rays will hit the variousreflecting surfaces 126 at different angles. Therefore, different lightrays will be reflected out of the substrate by different ones of thereflecting surfaces. The selection of which light rays will be reflectedout of the substrate by which surface 126 is engineered by selecting anappropriate angle of the surfaces 126. More details of a light-guideoptical element can be found in United States Patent Publication No.2008/0285140, entitled “Substrate-Guided Optical Devices,” published onNov. 20, 2008. In one embodiment, each eye will have its own light-guideoptical element 115. When the head mounted display device 2 has twolight-guide optical elements, each eye can have its own microdisplay 120that can display the same image in both eyes or different images in thetwo eyes. In another embodiment, there can be one light-guide opticalelement which reflects light into both eyes.

Mounted to or inside temple 102 is an image source, which (in oneembodiment) includes microdisplay 120 for projecting a virtual image andlens 122 for directing images from microdisplay 120 into the opticallens assembly 113. In one embodiment, lens 122 is a collimating lens.Microdisplay 120 projects an image through lens 122. There are differentimage generation technologies that can be used to implement microdisplay120. For example, microdisplay 120 can be implemented in using an OLED(Organic Light Emitting Diode) array, which is a self-emitting type ofspatial color microdisplay. Additionally, microdisplay 120 can beimplemented using a set of RGB light emitters and a scanning technology.For example, a PicoP™ display engine from Microvision, Inc. emits RGBlaser signals with a micro mirror steering either onto a tiny screenthat acts as a transmissive element or beamed directly into the eye.

Microdisplay 120 can also be implemented using a reflective technologyfor which external light is reflected and modulated by an opticallyactive material. The illumination is provided by either a white sourceor RGB source, depending on the technology. Digital light processing(DLP), liquid crystal on silicon (LCOS) and Mirasol® display technologyfrom Qualcomm, Inc. are examples of reflective display technologieswhich are efficient as most energy is reflected away from the modulatedstructure and may be used in the present system. Additionally,microdisplay 120 can be a transmissive projection technology where thelight source is modulated by optically active material, backlit withwhite light. These technologies are usually implemented using LCD typedisplays with powerful backlights and high optical energy densities.

Control circuits 136 provide various electronics that support the othercomponents of head mounted display device 2. More details of controlcircuits 136 are provided below with respect to FIG. 4. Inside ormounted to temple 102 are ear phones 130, inertial measurement unit 132and temperature sensor 138. In one embodiment shown in FIG. 4, theinertial measurement unit 132 (or IMU 132) includes inertial sensorssuch as a three axis magnetometer 132A, three axis gyro 132B and threeaxis accelerometer 132C. The inertial measurement unit 132 sensesposition, orientation, and sudden accelerations (pitch, roll and yaw) ofhead mounted display device 2. The IMU 132 may include other inertialsensors in addition to or instead of magnetometer 132A, gyro 132B andaccelerometer 132C.

Head mounted display device 2 also includes a system for tracking theposition of the user's eyes. The system may track the user's eyeposition and orientation so that the system can determine the FOV of theuser. However, a human will not perceive everything in front of them.Instead, a user's eyes will be directed at a subset of the environment.Therefore, in one embodiment, the system may include technology fortracking the position of the user's eyes in order to refine themeasurement of the FOV of the user.

For example, head mounted display device 2 may include eye trackingassembly 134 (FIG. 3), which has an eye tracking illumination device134A and eye tracking camera 134B (FIG. 4). In one embodiment, eyetracking illumination device 134A includes one or more infrared (IR)emitters, which emit IR light toward the eye. Eye tracking camera 134Bincludes one or more cameras that sense the reflected IR light. Theposition of the pupil can be identified by known imaging techniqueswhich detect the reflection of the cornea. For example, see U.S. Pat.No. 7,401,920, entitled “Head Mounted Eye Tracking and Display System”,issued Jul. 22, 2008. Such a technique can locate a position of thecenter of the eye relative to the tracking camera. Generally, eyetracking involves obtaining an image of the eye and using computervision techniques to determine the location of the pupil within the eyesocket. In one embodiment, it is sufficient to track the location of oneeye since the eyes usually move in unison. However, it is possible totrack each eye separately. Other eye tracking technologies are possible.

FIG. 3 only shows half of the head mounted display device 2. A full headmounted display device may include another optical lens assembly 113,another microdisplay 120, another lens 122, room-facing camera 112, eyetracking assembly 134, earphones, and temperature sensor.

FIG. 4 is a block diagram depicting the various components of headmounted display device 2. FIG. 5 is a block diagram describing thevarious components of processing unit 4. Head mounted display device 2,the components of which are depicted in FIG. 4, is used to provide avirtual experience to the user by fusing one or more virtual imagesseamlessly with the user's view of the real world. Additionally, thehead mounted display device components of FIG. 4 include many sensorsthat track various conditions. Head mounted display device 2 willreceive instructions about the virtual image from processing unit 4 andwill provide the sensor information back to processing unit 4.Processing unit 4 may determine where and when to provide a virtualimage to the user and send instructions accordingly to the head mounteddisplay device of FIG. 4.

Some of the components of FIG. 4 (e.g., room-facing camera 112, eyetracking camera 134B, microdisplay 120, opacity filter 114, eye trackingillumination 134A, earphones 130, and temperature sensor 138) are shownin shadow to indicate that there are two of each of those devices, onefor the left side and one for the right side of head mounted displaydevice 2. FIG. 4 shows the control circuit 200 in communication with thepower management circuit 202. Control circuit 200 includes processor210, memory controller 212 in communication with memory 214 (e.g.,D-RAM), camera interface 216, camera buffer 218, display driver 220,display formatter 222, timing generator 226, display out interface 228,and display in interface 230.

In one embodiment, the components of control circuit 200 are incommunication with each other via dedicated lines or one or more buses.In another embodiment, the components of control circuit 200 is incommunication with processor 210. Camera interface 216 provides aninterface to the two room-facing cameras 112 and stores images receivedfrom the room-facing cameras in camera buffer 218. Display driver 220will drive microdisplay 120. Display formatter 222 provides information,about the virtual image being displayed on microdisplay 120, to opacitycontrol circuit 224, which controls opacity filter 114. Timing generator226 is used to provide timing data for the system. Display out interface228 is a buffer for providing images from room-facing cameras 112 to theprocessing unit 4. Display in interface 230 is a buffer for receivingimages such as a virtual image to be displayed on microdisplay 120.Display out interface 228 and display in interface 230 communicate withband interface 232 which is an interface to processing unit 4.

Power management circuit 202 includes voltage regulator 234, eyetracking illumination driver 236, audio DAC and amplifier 238,microphone preamplifier and audio ADC 240, temperature sensor interface242 and clock generator 244. Voltage regulator 234 receives power fromprocessing unit 4 via band interface 232 and provides that power to theother components of head mounted display device 2. Eye trackingillumination driver 236 provides the IR light source for eye trackingillumination 134A, as described above. Audio DAC and amplifier 238output audio information to the earphones 130. Microphone preamplifierand audio ADC 240 provides an interface for microphone 110. Temperaturesensor interface 242 is an interface for temperature sensor 138. Powermanagement circuit 202 also provides power and receives data back fromthree axis magnetometer 132A, three axis gyro 132B and three axisaccelerometer 132C.

FIG. 5 is a block diagram describing the various components ofprocessing unit 4. FIG. 5 shows control circuit 304 in communicationwith power management circuit 306. Control circuit 304 includes acentral processing unit (CPU) 320, graphics processing unit (GPU) 322,cache 324, RAM 326, memory controller 328 in communication with memory330 (e.g., D-RAM), flash memory controller 332 in communication withflash memory 334 (or other type of non-volatile storage), display outbuffer 336 in communication with head mounted display device 2 via bandinterface 302 and band interface 232, display in buffer 338 incommunication with head mounted display device 2 via band interface 302and band interface 232, microphone interface 340 in communication withan external microphone connector 342 for connecting to a microphone, PCIexpress interface for connecting to a wireless communication device 346,and USB port(s) 348. In one embodiment, wireless communication device346 can include a Wi-Fi enabled communication device, BlueToothcommunication device, infrared communication device, etc. The USB portcan be used to dock the processing unit 4 to computing system 22 inorder to load data or software onto processing unit 4, as well as chargeprocessing unit 4. In one embodiment, CPU 320 and GPU 322 are the mainworkhorses for determining where, when and how to insert virtualthree-dimensional objects into the view of the user. More details areprovided below.

Power management circuit 306 includes clock generator 360, analog todigital converter 362, battery charger 364, voltage regulator 366, headmounted display power source 376, and temperature sensor interface 372in communication with temperature sensor 374 (possibly located on thewrist band of processing unit 4). Analog to digital converter 362 isused to monitor the battery voltage, the temperature sensor and controlthe battery charging function. Voltage regulator 366 is in communicationwith battery 368 for supplying power to the system. Battery charger 364is used to charge battery 368 (via voltage regulator 366) upon receivingpower from charging jack 370. HMD power source 376 provides power to thehead mounted display device 2.

FIG. 6 illustrates the alignment of the left and right optical lensassemblies 113L and 113R to each other and relative to theinterpupillary distance (IPD) of a user's left and right eyes 140L and140R. During manufacture of the head mounted display device 2, thealignment, or boresight, of the lens assemblies 113L and 113R may be setrelative to each other. In embodiments, the boresight of lens assemblies113L and 113R may be selected so that planar surfaces of the lensassemblies 113L and 113R are parallel to each other, and rays 143extending perpendicularly from the planar surfaces of lens assemblies113L and 113R extend to infinity without intersecting. It is understoodthat the boresight of lens assemblies 113L and 113R may be other thanparallel to each other in further embodiments, for example so that rays143 verge toward each other.

The lens assemblies 113L and 113R are also spaced from each other adistance equal to the IPD of most users, so that centers of the lensassemblies 113L and 113R aligns with centers of a user's eyes. IPDvaries among users, for example between 50 mm and 75 mm. Thus, the lensassemblies 113L and 113R may be spaced from each other, center-to-centersomewhere within this range.

Another optical property of the optical lens assemblies 113L and 113Rthat is set during manufacture of the head mounted display device 2 isthe focal distance of the lens assemblies 113L and 113R. FIG. 7 showsthe light-guide optical element 115L, 115R and focal plane lens 118L,118R of the respective lens assemblies 113L and 113R. In the leftoptical lens assembly 113L, light rays 141L received from thelight-guide optical element 115L are redirected by the focal plane lens118L so as to appear to the eye 140L of the user to come to a focus at afocal plane 145. Within tolerances, the right optical lens assembly 113Ris configured to focus images at the same focal plane 145.

Images from the left and right microdisplays may contain virtual objectswhich, when processed by the brain, appear to exist at different depthsrelative to the head mounted display device 2 (in front of or behind thefocal plane 145). However, the image including all virtual objectscreated by the left microdisplay and directed by focal plane lens 118Lfocuses at the focal plane 145. The same is true for the image createdby the right microdisplay and directed by the focal plane lens 118R.Different virtual depths are created by where the left and right virtualobjects converge relative to the focal plane, controlled by binoculardisparity of the virtual objects displayed by the left and rightmicrodisplays.

For example, the light rays 141L and 141R in FIG. 7 cause the user'seyes to converge inward in the directions of arrows 146L and 146R,respectively, to focus on the objects on focal plane 145. As shown inFIG. 8, when combined with the binocular disparity between the virtualobjects displayed to the left and right eyes, the brain interprets thevirtual object as existing at a given virtual depth, D_(v), which inthis example is farther than the distance to the focal plane, D_(fp).

FIG. 8 illustrates a virtual object 157 (a virtual cube in thisexample). The virtual object 157 is generated from a display of aportion 157L of an image onto the focal plane 145 by the leftmicrodisplay, and a display of a portion 157R of an image onto the focalplane by the right microdisplay. The portions 157L and 157R are shown ashaving depth on the focal plane 145 so they can be seen, but they wouldin fact be flat on the focal plane and not visible in FIG. 8. FIG. 9illustrates the left image 148L from the left microdisplay including theportion 157L, and FIG. 10 illustrates the right image 148R from theright microdisplay including the portion 157R. FIG. 8 contains onlyportions 157L/R of images 148L/R. As discussed below, the entire images148L/R would overlap each other as explained below.

FIG. 11 is an illustration of left and right images 148L and 148Rconventionally displayed at different focal distances. In examples, theleft and right images 148L and 148R both project outward horizontally.As can be seen, the spatial extent of displayed images 148L, 148R getslarger the further the image focal plane is from the head mounteddisplay device 2. This example mirrors the example shown in FIG. 6,where the boresight of the optical lens assemblies 113L and 113R extendin parallel rays out to infinity. Thus, the outer extents of thehorizontal angles run parallel to each other. Where the boresightextends in non-parallel rays, the outer extents of the left and rightimages would also be non-parallel with respect to each other.

As can be seen in FIG. 11, the images 148L and 148R overlap each other(shown with a darker area in FIG. 11). A first side band 150 existsincluding the left image 148L but not right image 148R, and a secondside band 152 exists including the right image 148R but not left image148L. The combined 3D image tends to display more dimly in the sidebands 150 and 152, and the side bands can also interfere with thestereopsis effect of 3D images.

Where boresight is set to infinity, the bands 150 and 152 have little orno effect on the displayed images at large distances. That is becausethe side bands have a constant width (in this example), but at largedistances, the overlap area becomes comparatively quite large. Forexample, focal plane 145-3 is set at 100 meters. At this distance, theoverlap of the left and right images 148L and 148R is above 99%.However, at closer distances in this example, the amount of overlapdecreases. At 2 m, the left and right images 148L, 148R focused on focalplane 145-2 have an overlap determined to be about 94.5%. And at 1 m,the left and right images 148L, 148R focused on focal plane 145-1 havean overlap determined to be about 89.4%. FIG. 12 is a conventional graphof the amount of horizontal overlap versus distance for two differentinterocular distances (52.5 mm and 62.5 mm). As can be seen, at smalldistances, the amount of overlap decreases and the side bands 150 and152 where the images do not overlap become large relative to theoverlapped sections.

In accordance with aspects of the present technology, the overlap of theleft and right images 148L and 148R may be shifted toward each other tohave complete or substantially complete overlap at some predefineddistance from the head mounted display assembly. This predefineddistance may be the focal distance but need not be in furtherembodiments.

In embodiments, the display shift algorithm for shifting of the left andright images 148L and 148R may be performed in a one-time calibrationprocess during manufacture of the head mounted display device 2.However, in addition to or instead of the manufacturing calibrationprocess, the present technology may implement the display shiftalgorithm dynamically in the runtime rendering process during use of thehead mounted display device 2. Implementations of the display shiftalgorithm as a one-time manufacturing calibration process and in theruntime rendering process are explained below.

FIG. 13 illustrates a flowchart for implementing the display shiftalgorithm as a one-time calibration process during manufacture of thehead mounted display device 2. In step 400, boresight of the respectiveoptical lens assemblies 113L and 113R is set. Step 400 is performedduring assembly and mounting of the optical lens assemblies into thehead mounted display device.

In step 402, the focal distance for a predefined focal plane may be set.This focal distance may in fact be set when the optical lens assembly isconfigured and assembled, and as explained above, may at least in partbe set by the concavity of the focus lenses 118L and 118R of the opticallens assemblies 113. The head mounted display device 2 may bemanufactured with any predesigned focal distance. It has been determinedthat the eyes verge comfortably to a distance of 2 meters, and forgeneral use embodiments, the focal distance may be manufactured intohead mounted display device at 2 meters.

However, certain head mounted display devices may be manufactured forspecialized applications. For example, some professions such ascommercial and military airplane pilots focus on the horizon whenflying. Head mounted displays manufactured for such specialized use mayhave their focal distance set at infinity. Other professions, such assurgeons, typically focus at distances shorter than 2 meters, forexample 1 meter, when operating. Head mounted displays manufactured forsuch specialized use may have their focal distance set at 1 meter. Otherfocal distances are contemplated.

In step 406, the left and right image overlap may be determined for somedesired, predefined distance. As noted, this distance may typically bethe focal plane distance, but need not be in further embodiments. Also,the left and right image overlap may be the horizontal left and rightimage overlap. However, as noted below, the overlap may alternatively oradditionally be the vertical left and right image overlap in furtherembodiments. As noted in the graph of FIG. 12, the horizontal stereooverlap of the left and right images is a known function of distance,depending on the spacing between the optical lens assemblies 113L and113R, and on the boresight set for the optical lens assemblies 113L and113R. Vertical overlap of the left and right images is also a knownfunction of distance, depending on the parallelism of the respectiveoptical lens assemblies about a horizontal axis through the lensassemblies.

The amount of horizontal and/or vertical image overlap in step 406 maybe measured in pixels. As shown in FIG. 14, knowing the focal distanceand boresight, a processor may determine the pixels where the left andright images are to be displayed by the left and right microdisplays120. The processor may be processing unit 4, or some other processorused during manufacture of the head mounted display device 2. The leftand images may each be the same size and may be a predefined number ofrows and columns. In embodiments, the pixel display positions of theleft and right images may be described in terms of an origin pixel.Different origin pixels may be used, but in embodiments, the originpixel may be the pixel at the upper left of the displayed image pixelgrid.

Thus, in FIG. 14, the origin pixel of the left image 148L is at pixelcoordinate (x1L, y1L). The origin pixel of the right image 148R is atpixel coordinate (x1R, y1R). The degree of horizontal overlap correctionof the left and right images is then given by the difference between x1Land x1R. By simple geometry, the angle subtended by each microdisplaypixel can be calculated. Then shifting each of the left and right imageshorizontally by the determined number of pixels would provide completehorizontal overlap of the left and right images. The shift direction isdependent on the orientation and design of the optics that follow themicrodisplay, as is well known in the art of boresight alignment.Similarly, the degree of vertical overlap correction of the left andright images is then given by the difference between y1L and y1R.Shifting each of the left and right images vertically by the determinednumber of pixels can provide complete vertical overlap of the left andright images.

In step 408, the display shift algorithm determines whether there areavailable border pixels to make the horizontal and/or vertical shift. Inparticular, the pixels used for displaying the left and right images148L, 148R (i.e., “active pixels”) may be smaller than the overall fieldof pixels (i.e., “native pixels”) available to the left and rightmicrodisplays 120. As shown in FIG. 14, a border 154 of available pixelsmay be provided which are unused for the display of images 148L, 148R.In one example, the images 148L and 148R may use a grid of active pixels1920 by 1080, where the overall field of native pixels may be 2000 by1160. Thus, assuming the images 148L and 148R are initially centered inthe field of native pixels, each image may shift horizontally andvertically:

Shift_(max) _(_) _(h)=(2000−1920)/2=40 pixels

Shift_(max) _(_) _(v)=(1160−1080)/2=40 pixels.

It is understood that the above number of active and native pixels usedfor images 148L, 148R are by way of example only and the number ofactive and native pixels in the images may vary, proportionately ordisproportionately, above or below these numbers in further examples. Itis noted that both the left and right images 148L, 148R may shift, sothe overall horizontal or vertical shift of the images relative to eachother may be twice the maximum shift numbers set forth above.

Referring again to step 408, the display shift algorithm checks whetherthere are sufficient border pixels to shift the image horizontallyand/or vertically the number of pixels determined in step 406 so thatthe images 148L and 148R completely overlap. If so, the image is shiftedhorizontally and/or vertically in step 410. As shown in FIG. 15, theimages 148L and 148R may be shifted equal amounts horizontally, forexample by resetting the origin pixel in the left image 148L from (x1L,y1L) to (x2L, y1L), and by resetting the origin pixel in the right image148R from (x1R, y1R) to (x2R, y1R). It is understood that the originpixels may be adjusted similarly for a vertical shift.

If, on the other hand, it is determined in step 408 that there are notenough pixels in the border 154 to shift an amount to complete theoverlap of the left and right images, the images may be shifted in step414 by the amount of pixels that are available in the border 154.

After the one-time calibration process shown in FIG. 13, the displayedleft and right images 148L and 148R may thereafter display on thedefined focal plane. For example, as shown in FIG. 16, where the focalplane 145 is set to 2 m, the calibration process of FIG. 13 allowscomplete horizontal stereo overlap of the left and right images 148L and148R.

While less than complete vertical overlap is not a function of theboresight selection for the optical lens assemblies 113, it may happenthat there is less than complete vertical overlap due for example tomanufacturing tolerances. For this purpose, the display shift algorithmof FIG. 13 may be run after the optical lens assemblies 113L and 113Rare assembled to correct for any vertical mismatch and lack of completevertical overlap.

As noted, instead of or in addition to the one-time calibration processat manufacturing of the head mounted display device 2, the display shiftalgorithm may be implemented dynamically in the runtime renderingprocess during use of the head mounted display device 2. As one possiblereason for doing this, it is conceivable that, instead of having a fixedfocal plane, embodiments of the head mounted display device may have adynamic focal plane. For example, different software applications usedwith the head mounted display device 2 may set a focal plane distancethat is optimized for that application. As noted above, some specializeduses include flying or surgery, and a single head mounted display device2 may be used across these disciplines using a focal plane that may beset by an application running on the processing unit 4.

FIG. 17 is a high level flowchart of the operation and interactivity ofthe processing unit 4 and head mounted display device 2 during adiscrete time period in the use of the head mounted display device 2.The discrete time period may for example be the time it takes togenerate, render and display a single frame of image data to a user. Inembodiments, data may be refreshed at a rate of 60 Hz, though it may berefreshed more often or less often in further embodiments.

The system for presenting a virtual environment to one or more users maybe configured in step 600. As noted above, a user's IPD may affect thedegree to which the left and right images overlap. Thus, step 600 mayinclude determining a user's IPD, and automatically adjusting the imagerendering stereo separation for optimal user viewing. In some systems,the optical lens assemblies may also be adjusted for IPD either using asoftware algorithm or mechanical adjustment of the optical lensassemblies. Techniques for automatically determining a user's IPD andautomatically adjusting the optical lens assemblies to set the IPD foroptimal user viewing, are discussed in U.S. Pat. No. 8,487,838, entitled“Gaze Detection In A See-Through, Near-Eye, Mixed Reality Display”; U.S.Pat. No. 9,025,252, entitled “Adjustment Of A Mixed Reality Display ForInter-Pupillary Distance Alignment”; and U.S. patent application Ser.No. 13/221,662 entitled “Aligning Inter-Pupillary Distance In A Near-EyeDisplay System.”

In step 600, the processing unit 4 may also receive a dynamically setfocal plane distance. The dynamic setting of focal plane distance isbeyond the present scope. However, once the focal plane distance is set,the overlap of displayed left and right images at that distance may becontrolled as explained above and below.

In steps 604 the processing unit 4 gathers data from the scene. This maybe image data sensed by the head mounted display device 2, and inparticular, by the room-facing cameras 112, the eye tracking assemblies134 and the IMU 132. A scene map may be developed in step 610identifying the geometry of the scene as well as the geometry andpositions of objects within the scene. In embodiments, the scene mapgenerated in a given frame may include the x, y and z positions of auser's hand(s), other real world objects and virtual objects in thescene.

In step 612, the system may detect and track a user's skeleton and/orhands, and update the scene map based on the positions of moving bodyparts and other moving objects. In step 614, the processing unit 4determines the x, y and z position, the orientation and the FOV of thehead mounted display device 2 within the scene. Further details of step614 are now described with respect to the flowchart of FIG. 18.

In step 700, the image data for the scene is analyzed by the processingunit 4 to determine both the user head position and a face unit vectorlooking straight out from a user's face. The head position may beidentified from feedback from the head mounted display device 2, andfrom this, the face unit vector may be constructed. The face unit vectormay be used to define the user's head orientation and, in examples, maybe considered the center of the FOV for the user. The face unit vectormay also or alternatively be identified from the camera image datareturned from the room-facing cameras 112 on head mounted display device2. In particular, based on what the cameras 112 on head mounted displaydevice 2 see, the processing unit 4 is able to determine the face unitvector representing a user's head orientation.

In step 704, the position and orientation of a user's head may also oralternatively be determined from analysis of the position andorientation of the user's head from an earlier time (either earlier inthe frame or from a prior frame), and then using the inertialinformation from the IMU 132 to update the position and orientation of auser's head. Information from the IMU 132 may provide accurate kinematicdata for a user's head, but the IMU typically does not provide absoluteposition information regarding a user's head. This absolute positioninformation, also referred to as “ground truth,” may be provided fromthe image data obtained from the cameras on the head mounted displaydevice 2.

In embodiments, the position and orientation of a user's head may bedetermined by steps 700 and 704 acting in tandem. In furtherembodiments, one or the other of steps 700 and 704 may be used todetermine head position and orientation of a user's head.

It may happen that a user is not looking straight ahead. Therefore, inaddition to identifying user head position and orientation, theprocessing unit may further consider the position of the user's eyes inhis head. This information may be provided by the eye tracking assembly134 described above. The eye tracking assembly is able to identify aposition of the user's eyes, which can be represented as an eye unitvector showing the left, right, up and/or down deviation from a positionwhere the user's eyes are centered and looking straight ahead (i.e., theface unit vector). A face unit vector may be adjusted to the eye unitvector to define where the user is looking.

In step 710, the FOV of the user may next be determined. The range ofview of a user of a head mounted display device 2 may be predefinedbased on the up, down, left and right peripheral vision of ahypothetical user. In order to ensure that the FOV calculated for agiven user includes objects that a particular user may be able to see atthe extents of the FOV, this hypothetical user may be taken as onehaving a maximum possible peripheral vision. Some predetermined extraFOV may be added to this to ensure that enough data is captured for agiven user in embodiments. The FOV for the user at a given instant maythen be calculated by taking the range of view and centering it aroundthe face unit vector, adjusted by any deviation of the eye unit vector.

Referring again to FIG. 7, in step 618, the processing unit 4 may cullthe rendering operations so that just those virtual objects which couldpossibly appear within the final FOV of the head mounted display device2 are rendered. The positions of other virtual objects may still betracked, but they are not rendered. It is also conceivable that, infurther embodiments, step 618 may be skipped altogether and the entireimage is rendered.

The processing unit 4 may next perform a rendering setup step 620 wheresetup rendering operations are performed using the scene map and FOVreceived in steps 610 and 614. Once virtual object data is received, theprocessing unit may perform rendering setup operations in step 620 forthe virtual objects which are to be rendered in the FOV. The setuprendering operations in step 620 may include common rendering tasksassociated with the virtual object(s) to be displayed in the final FOV.These rendering tasks may include for example, shadow map generation,lighting, and animation. In embodiments, the rendering setup step 620may further include a compilation of likely draw information such asvertex buffers, textures and states for virtual objects to be displayedin the predicted final FOV.

In accordance with aspects of the present technology, the renderingsetup operations of step 620 may further execute the display shiftalgorithm to determine the horizontal and/or vertical shift of thedisplayed images 148L and 148R for a defined distance, such as the focaldistance currently being used by the head mounted display device.Details of the display shift algorithm executed at runtime renderingsetup step 620 may be the same as explained above with respect to thesteps 406-414 of the flowchart of FIG. 13, and the views of FIGS. 14-16.While the display shift algorithm is described as being performed aspart of the rendering setup operations 620, it is understood that thedisplay shift algorithm may be performed at any point during theflowchart of FIG. 17 after the distance, such as the focal distance, isset.

Using the information regarding the locations of objects in the 3-Dscene map, the processing unit 4 may next determine occlusions andshading in the user's FOV in step 624. In particular, the scene map hasx, y and z positions of objects in the scene, including any moving andnon-moving virtual or real objects. Knowing the location of a user andtheir line of sight to objects in the FOV, the processing unit 4 maythen determine whether a virtual object partially or fully occludes theuser's view of a real world object. Additionally, the processing unit 4may determine whether a real world object partially or fully occludesthe user's view of a virtual object.

In step 626, the GPU 322 of processing unit 4 may next render images148L and 148R to be displayed to the user, with the overlap adjusted asexplained above. Portions of the rendering operations may have alreadybeen performed in the rendering setup step 620 and periodically updated.Any occluded virtual objects may not be rendered, or they may berendered. Where rendered, occluded objects will be omitted from displayby the opacity filter 114 as explained above.

In step 630, the processing unit 4 checks whether it is time to send arendered image to the head mounted display device 2, or whether there isstill time for further refinement of the image using more recentposition feedback data from the head mounted display device 2. In asystem using a 60 Hertz frame refresh rate, a single frame is about 16ms.

If time to display images 184L and 184R, the images are sent tomicrodisplay 120 to be displayed at the appropriate pixels, accountingfor adjusted overlap, perspective and occlusions. At this time, thecontrol data for the opacity filter is also transmitted from processingunit 4 to head mounted display device 2 to control opacity filter 114.The head mounted display would then display the images 184L and 184R tothe user in step 634.

On the other hand, where it is not yet time to send a frame of imagedata to be displayed in step 630, the processing unit may loop back formore recent sensor data to refine the predictions of the final FOV andthe final positions of objects in the FOV. In particular, if there isstill time in step 630, the processing unit 4 may return to step 604 toget more recent sensor data from the head mounted display device 2.

The processing steps 600 through 634 are described above by way ofexample only. It is understood that one or more of these steps may beomitted in further embodiments, the steps may be performed in differingorder, or additional steps may be added.

In embodiments described above, a partial overlap of displayed left andright images may be adjusted at the left and right microdisplays 120using software to substantially or completely overlap at a givendistance such as the focal distance. In further embodiments, it iscontemplated that the left and right microdisplays 120 may be mountedfor translation on mechanical actuators. In such embodiments, instead ofshifting the left and right images on the microdisplays using software,the microdisplays themselves may be shifted, under control of theprocessing unit 4, to achieve substantial or complete overlap of thedisplayed left and right images at the desired distance. Additionally,moveable transmissive optical elements, such as glass wedges, may beplaced in the microdisplay image path to introduce lateral and verticalshifts in the image position.

Additionally, embodiments described above provide a method forincreasing the overlap of displayed left and right images to reduce orremove side bands which may appear dimmer. However, it is conceivable toprovide a system where it is desirable to decrease overlap of thedisplayed left and right images at a certain predefined distance. Thepresent technology may operate to decrease overlap at a predefineddistance using the system and method described above.

Embodiments of the present technology may be implemented on computerreadable media. The computer readable media may be a computer readablesignal medium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, or semiconductor system, apparatus,or device, or any suitable combination of the foregoing. More specificexamples (a non-exhaustive list) of the computer readable storage mediumwould include the following: a portable computer diskette, a hard disk,a random access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an appropriateoptical fiber with a repeater, a portable compact disc read-only memory(CD-ROM), an optical storage device, a magnetic storage device, or anysuitable combination of the foregoing. In the context of this document,a computer readable storage medium may be any tangible medium that cancontain, or store a program for use by or in connection with aninstruction execution system, apparatus, or device. A computer readablestorage medium does not include transitory, modulated or other types ofsignals.

In summary, an example of the present technology relates to a system forcontrolling an amount of overlap of displayed left and right images, thedisplayed left and right images presenting a virtual environment,comprising: a head mounted display device including one or more displaysfor displaying the left and right images; and a processing unitoperatively coupled to the display device, the processing unitconfigured to shift the displayed left image and to shift the displayedright image at least one of horizontally and vertically, to alter atleast one of a horizontal and vertical overlap of the left and rightimages at a defined distance from the head mounted display device.

Another example of the present technology relates to a method ofdisplaying left and right images in a system displaying virtual images,the system comprising at least one display for displaying the left andright images and left and right optical lens assemblies through whichthe left and right images are displayed, method comprising: (a)determining a focal distance at which the left and right opticalassemblies focus the left and right images; (b) determining an overlapof the left and right images at the focal distance; and (c) horizontallyshifting the left and right images displayed by the at least one displayto align horizontally with each other at the focal distance.

In a further example, the present technology relates to a computerreadable storage medium for controlling a processor to perform a methodof displaying left and right images in a system displaying virtualimages, method comprising: (a) configuring first and second displays todisplay first and second images, the first and second image creating a3D stereopsis effect when viewed by left and right eyes, the firstdisplay having an overall field of native pixels that is larger than thefirst image so that a border of pixels exists at at least one edge ofactive pixels used to display the first image, the border of pixels notused to display the first image; (b) determining a distance at which toshift the first and second images to greater overlap each other; (c)determining an overlap of the first and second images at the distance;and (d) shifting the first image into the border of pixels to overlap toa greater degree with the second image at the distance.

In another example, the present technology relates to a means forcontrolling an amount of overlap of displayed left and right images, thedisplayed left and right images presenting a virtual environment,comprising: means for displaying the left and right images; andprocessing means operatively coupled to the display means, theprocessing means shifting the displayed left image and shifting thedisplayed right image at least one of horizontally and vertically, toalter at least one of a horizontal and vertical overlap of the left andright images at a defined distance from the display means.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims. It is intended that the scopeof the invention be defined by the claims appended hereto.

We claim:
 1. A system for controlling an amount of overlap of displayedleft and right images, the displayed left and right images presenting avirtual environment, comprising: a head mounted display device includingone or more displays for displaying the left and right images; and aprocessing unit operatively coupled to the display device, theprocessing unit configured to shift the displayed left image and toshift the displayed right image at least one of horizontally andvertically, to alter at least one of a horizontal and vertical overlapof the left and right images at a defined distance from the head mounteddisplay device.
 2. The system of claim 1, wherein the processing unitshifts the left and right displayed images to overlap each other at thepredefined distance.
 3. The system of claim 1, the left displayed imagebeing part of an overall field of native pixels, the overall field ofnative pixels including a border of pixels around the left displayedimage and unused by the displayed image prior to the left image beingshifted.
 4. The system of claim 3, the processing unit shifting the leftimage into the border of pixels around the left displayed image.
 5. Thesystem of claim 1, wherein the predefined distance is a focal distanceat which the left and right displayed images are focused.
 6. The systemof claim 1, further comprising left and right optical lens assembliesthrough which light from the left and right images passes, at least onelens within at least one of the left and right optical lens assemblydefining a focal distance at which at least one of the left and rightdisplayed images are focused.
 7. The system of claim 6, wherein anamount by which the processing unit shifts the displayed left image andshifts the displayed right image is dependent on a spacing by which theleft and right optical lens assemblies are separated.
 8. The system ofclaim 6, wherein the predefined distance is a focal distance.
 9. Thesystem of claim 1, wherein the predefined distance changes withdifferent software applications run by the processing unit, wherein theamount by which the left image is shifted and the amount by which theright image is shifted changes with a change in the predefined distance.10. A method of displaying left and right images in a system displayingvirtual images, the system comprising at least one display fordisplaying the left and right images and left and right optical lensassemblies through which the left and right images are displayed, methodcomprising: (a) determining a focal distance at which the left and rightoptical assemblies focus the left and right images; (b) determining anoverlap of the left and right images at the focal distance; and (c)horizontally shifting the left and right images displayed by the atleast one display to align horizontally with each other at the focaldistance.
 11. The method of claim 10, further comprising the step ofvertically shifting the left and right images displayed by the at leastone display to align vertically with each other at the focal distance.12. The method of claim 10, further comprising the step of providing aborder of pixels around active pixels used to display the left image,the left image horizontally shifting into the border of pixels.
 13. Themethod of claim 10, wherein said steps (a), (b) and (c) are performed ata time when the at least one display and the left and right optical lensassemblies are assembled together.
 14. The method of claim 10, whereinsaid step (c) is performed at a time when the left and right images arerendered during use of the system displaying virtual images.
 15. Themethod of claim 10, further comprising the step of varying the focaldistance after a time when the at least one display and the left andright optical lens assemblies have been assembled together.
 16. Themethod of claim 10, wherein said step (c) is performed by a mechanicalactuator shifting the at least one display.
 17. The method of claim 10,wherein said step (c) is performed by a software algorithm executed by aprocessor associated with the at least one display.
 18. A computerreadable storage medium for controlling a processor to perform a methodof displaying left and right images in a system displaying virtualimages, method comprising: (a) configuring first and second displays todisplay first and second images, the first and second image creating a3D stereopsis effect when viewed by left and right eyes, the firstdisplay having an overall field of native pixels that is larger than thefirst image so that a border of pixels exists at at least one edge ofactive pixels used to display the first image, the border of pixels notused to display the first image; (b) determining a distance at which toshift the first and second images to greater overlap each other; (c)determining an overlap of the first and second images at the distance;and (d) shifting the first image into the border of pixels to overlap toa greater degree with the second image at the distance.
 19. The computerreadable storage medium of claim 18, said step (b) comprising the stepof setting a focal distance, the focal distance being the distance atwhich the first and second images are shifted to overlap each other. 20.The computer readable storage medium of claim 17, said step (d)comprising the step of shifting the first image to completely overlapwith the second image, the second image shifted in an equal and oppositedirection, in the event there are enough pixels in the border tocompletely shift the first image.